Repository: EleutherAI/lm-evaluation-harness
Issue: How to use local downloaded mmlu dataset？
Root cause: Documentation Deficiency

Hi,

I have a try to use a local downloaded mmlu dataset. And refer to https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets and searched the issuse. but I still encounter some errors.

Then I changed the _default_config

```
dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0
```
I only changed the dataset_path and add a dataset_kwarg like following:
```
dataset_path: mmlu
dataset_kwargs:
  data_dir: mmlu_local/
```
but I still encontered the following error:
```
root@IP:/home/# lm_eval --tasks mmlu --model hf --model_args pretrained=/home/public_data/llama_2/llama-2-7b-hf/,trust_remote_code=True --device cuda:6  --batch_size 1 --num_fewshot 1 --gen_kwargs temperature=0,top_k=0,top_p=0 --verbosity DEBUG
2024-04-02:06:35:17,058 WARNING  [loading.py:546] Using the latest cached version of the module from /root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/9d3b67e0c429cd7460b2b05aab53419b48eea369b73e1d9f185a56ca90c373d4 (last modified on Fri Mar 29 07:47:57 2024) since it couldn't be found locally at evaluate-metric--exact_match, or remotely on the Hugging Face Hub.
2024-04-02:06:35:17,119 INFO     [__main__.py:251] Verbosity set to DEBUG
2024-04-02:06:35:24,007 INFO     [__main__.py:335] Selected Tasks: ['mmlu']
2024-04-02:06:35:24,009 INFO     [evaluator.py:131] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-04-02:06:35:24,009 WARNING  [evaluator.py:142] generation_kwargs specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!
2024-04-02:06:35:24,009 INFO     [evaluator.py:177] Initializing hf model, with arguments: {'pretrained': '/home/public_data/llama_2/llama-2-7b-hf/', 'trust_remote_code': True}
2024-04-02:06:35:24,013 WARNING  [logging.py:61] Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-04-02:06:35:24,013 INFO     [huggingface.py:163] Using device 'cuda:6'
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.81s/it]
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/lm-evaluation-harness/lm_eval/__main__.py", line 341, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/lm-evaluation-harness/lm_eval/evaluator.py", line 209, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 420, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 270, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 253, in _load_individual_task_or_group
    **dict(collections.ChainMap(*map(fn, subtask_list))),
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 253, in _load_individual_task_or_group
    **dict(collections.ChainMap(*map(fn, subtask_list))),
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 161, in _load_individual_task_or_group
    return load_task(task_config, task=name_or_config, group=parent_name)
  File "/home/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 150, in load_task
    task_object = ConfigurableTask(config=config)
  File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 782, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 874, in download
    self.dataset = datasets.load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1812, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'mmlu' on the Hub (ConnectTimeout)
```
Could someone give me some suggestions?

Hi!

I believe setting the environment variable `HF_DATASETS_OFFLINE="1"` would resolve this.

You may also need to change `dataset_name` to `parquet` if using the parquet data from this dataset repo.

Let me know if the above 2 points don't fix this!