Repository: EleutherAI/lm-evaluation-harness
Issue: How to load the local MMLU dataset
Root cause: Documentation Deficiency

In fact, I have download all the files in https://huggingface.co/datasets/hails/mmlu_no_train/tree/main. And I put it to `./hails/mmlu_no_train`.

I want to know how do I change the `lm-evaluation-harness/lm_eval/tasks/mmlu/default/_default_template_yaml` content?
I have tried many times and seek for solution in the issues, but don't get any right solution.

```
dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
dataset_kwargs:
  trust_remote_code: true

```

try cloning this one: https://huggingface.co/datasets/cais/mmlu and then you can set the `dataset_path` to the full path of the repository root, for ex:
`dataset_path: /Users/admin/Downloads/mmlu`

> try cloning this one: https://huggingface.co/datasets/cais/mmlu and then you can set the `dataset_path` to the full path of the repository root, for ex: `dataset_path: /Users/admin/Downloads/mmlu`

Thanks for your help.
But after the git clone and changing the `dataset_path`, when I run:
```
lm_eval \
    --model vllm \
    --model_args pretrained=/Pretrained_models/Qwen/Qwen2-7B-Instruct/,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=8 \
    --tasks mmlu \
    --batch_size 1
```
there is a error:
```
INFO     [__main__.py:279] Verbosity set to INFO
INFO     [__main__.py:376] Selected Tasks: ['mmlu']
INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
INFO     [evaluator.py:201] Initializing vllm model, with arguments: {'pretrained': '/Pretrained_models/Qwen/Qwen2-7B-Instruct/', 'tensor_parallel_size': 1, 'dtype': 'auto', 'gpu_memory_utilization': 0.8, 'data_parallel_size': 8}
WARNING  [vllm_causallms.py:105] You might experience occasional issues with model weight downloading when data_parallel is in use. To ensure stable performance, run with data_parallel_size=1 until the weights are downloaded and cached.
INFO     [vllm_causallms.py:110] Manual batching is not compatible with data parallelism.
Generating test split:   0%|                                                                                                 | 0/324 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 82, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 317, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1514, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/miniconda3/envs/lm_eval/bin/lm_eval", line 10, in <module>
    sys.exit(cli_evaluate())
  File "/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/lm-evaluation-harness/lm_eval/api/task.py", line 819, in __init__
    self.download(self.config.dataset_kwargs)
  File "/lm-evaluation-harness/lm_eval/api/task.py", line 926, in download
    self.dataset = datasets.load_dataset(
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/builder.py", line 1000, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/builder.py", line 1741, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/builder.py", line 1897, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
```
Do you have any clues about this problem? Thanks.

Maybe a corrupted file? The arguments get passed [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/4259a6d4b23c87e6cc9210650f662acf14e40b9b/lm_eval/api/task.py#L926) for reference, so you can pass any arg accepted by [load_dataset](https://huggingface.co/docs/datasets/en/loading) like so:
```yaml
dataset_path: ...
dataset_name: ...
dataset_kwargs:
  trust_remote_code: true
#  any other kwargs accepted by load_dataset
```
MMLU is spilt into multiple subsets so each one will be loaded with:
```python
load_dataset(path="/Users/admin/Downloads/mmlu", name="econometrics")
```
hope this helps!


It was indeed caused by the corrupted file. It's normal now, thanks for your kind help!