Repository: EleutherAI/lm-evaluation-harness
Issue: dataset path
Root cause: Documentation Deficiency

Hi, What should I do if I want to use the local dataset? I cannot reach "ceval/ceval-exam", I download the dataset to my local location, but I don't know how to set the local path.
Please help me. Thanks 

I also have the same issue with **loading local/offline benchmarks for evaluation**. Please help us. 🙏 Thank you. : )

Hello! 

Please see the following docs section for a few reference options on loading local datasets: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets

If there are any problems doing this in the way documented here please let me know! : ) 

I tried to eval model on mmlu following the docs. I met the following error: (connectionError)
```
ConnectionError: Couldn't reach https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /datasets/cais/mmlu/resolve/main/data.tar (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb3ab117dc0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))
```
the loacl dataset cannot be loaded successfully when datasets.load_dataset.

the yaml file:
```
dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
dataset_kwargs:
  data_dir: /workspace/data/mmlu_no_train/data
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0
``` 
The data was downloaded from https://huggingface.co/datasets/hails/mmlu_no_train/tree/main.
How should I set the path/dir/name of mmlu?
BTW, what's the meaning of the flag **sampler: first_n** ? how to eval in zero shot manner?

`sampler: first_n` means to take (in order) the first N samples from the fewshot split as fewshot examples! MMLU has a specific order the fewshot samples are typically presented in.


re: the MMLU local dataset, if you've downloaded it locally, you may have to do the following:
```
dataset_path: parquet
dataset_kwargs:
  data_dir: /workspace/data/mmlu_no_train/data
```
where data_dir is the path to the folder that has these 3 parquet files in it: https://huggingface.co/datasets/hails/mmlu_no_train/tree/main/all

> `sampler: first_n` means to take (in order) the first N samples from the fewshot split as fewshot examples! MMLU has a specific order the fewshot samples are typically presented in.
> 
> re: the MMLU local dataset, if you've downloaded it locally, you may have to do the following:
> 
> ```
> dataset_path: parquet
> dataset_kwargs:
>   data_dir: /workspace/data/mmlu_no_train/data
> ```
> 
> where data_dir is the path to the folder that has these 3 parquet files in it: https://huggingface.co/datasets/hails/mmlu_no_train/tree/main/all

We revise the _default_template_yaml to:

dataset_path: parquet
dataset_kwargs:
  data_dir: /home/liuhongyuan/lm-evaluation-harness/datasets/mmlu_no_train
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0


still got a problem 
  File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 810, in __init__                                                                                                                  
    if self.fewshot_docs() is not None:                                                               
       ^^^^^^^^^^^^^^^^^^^                                                                            
  File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 924, in fewshot_docs                                                                                                              
    return self.dataset[self.config.fewshot_split]                                                    
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                    
  File "/home/anaconda3/envs/llama/lib/python3.11/site-packages/datasets/dataset_dict.py", line 75, in __getitem__
    return super().__getitem__(k)                                                                     
           ^^^^^^^^^^^^^^^^^^^^^^                                                                     
KeyError: 'dev'              

Could you share what the contents of this mmlu_no_train folder look like perhaps?

> > `sampler: first_n` means to take (in order) the first N samples from the fewshot split as fewshot examples! MMLU has a specific order the fewshot samples are typically presented in.
> > re: the MMLU local dataset, if you've downloaded it locally, you may have to do the following:
> > ```
> > dataset_path: parquet
> > dataset_kwargs:
> >   data_dir: /workspace/data/mmlu_no_train/data
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > where data_dir is the path to the folder that has these 3 parquet files in it: https://huggingface.co/datasets/hails/mmlu_no_train/tree/main/all
> 
> We revise the _default_template_yaml to:
> 
> dataset_path: parquet dataset_kwargs: data_dir: /home/liuhongyuan/lm-evaluation-harness/datasets/mmlu_no_train test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: `"{{question.strip()}}\nA.` {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> 
> * metric: acc
>   aggregation: mean
>   higher_is_better: true
>   metadata:
>   version: 0.0
> 
> still got a problem File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 810, in **init** if self.fewshot_docs() is not None: ^^^^^^^^^^^^^^^^^^^ File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 924, in fewshot_docs return self.dataset[self.config.fewshot_split] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/anaconda3/envs/llama/lib/python3.11/site-packages/datasets/dataset_dict.py", line 75, in **getitem** return super().**getitem**(k) ^^^^^^^^^^^^^^^^^^^^^^ KeyError: 'dev'

I found a solution to the issue of not being able to connect to huggingface.co without needing to download the dataset locally. You just need to modify the corresponding _URL link in the dataset's Python script to a mirror site. 
For example, if the error message is:
```
File "YourPath/mmlu_no_train.py", line 129, in _split_generators
    archive = dl_manager.download(_URL)
```
Then reach to mmlu_no_train.py, and change 
```
_URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"
```
 to 
```
_URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"
```
The same applies to other datasets.

Is it possible to use local datasets such as bbh? For bbh I cannot find local files on huggingface and I need to download every subset instead of using one line of code to download the dataset. Could you show me the correct way to deal with such local datasets in the documents? Thanks. @haileyschoelkopf 

> `sampler: first_n` means to take (in order) the first N samples from the fewshot split as fewshot examples! MMLU has a specific order the fewshot samples are typically presented in.
> 
> re: the MMLU local dataset, if you've downloaded it locally, you may have to do the following:
> 
> ```
> dataset_path: parquet
> dataset_kwargs:
>   data_dir: /workspace/data/mmlu_no_train/data
> ```
> 
> where data_dir is the path to the folder that has these 3 parquet files in it: https://huggingface.co/datasets/hails/mmlu_no_train/tree/main/all

Sorry, your code will get `KeyError: 'dev'`, the corret code for local `mmlu` dataset should be：
```
dataset_path: autodl-tmp/mmlu_no_train
dataset_kwargs:
  data_dir: all
...
```




> > > `sampler: first_n` means to take (in order) the first N samples from the fewshot split as fewshot examples! MMLU has a specific order the fewshot samples are typically presented in.
> > > re: the MMLU local dataset, if you've downloaded it locally, you may have to do the following:
> > > ```
> > > dataset_path: parquet
> > > dataset_kwargs:
> > >   data_dir: /workspace/data/mmlu_no_train/data
> > > ```
> > > 
> > > 
> > >     
> > >       
> > >     
> > > 
> > >       
> > >     
> > > 
> > >     
> > >   
> > > where data_dir is the path to the folder that has these 3 parquet files in it: https://huggingface.co/datasets/hails/mmlu_no_train/tree/main/all
> > 
> > 
> > We revise the _default_template_yaml to:
> > dataset_path: parquet dataset_kwargs: data_dir: /home/liuhongyuan/lm-evaluation-harness/datasets/mmlu_no_train test_split: test fewshot_split: dev fewshot_config: sampler: first_n output_type: multiple_choice doc_to_text: `"{{question.strip()}}\nA.` {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:" doc_to_choice: ["A", "B", "C", "D"] doc_to_target: answer metric_list:
> > 
> > * metric: acc
> >   aggregation: mean
> >   higher_is_better: true
> >   metadata:
> >   version: 0.0
> > 
> > still got a problem File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 810, in **init** if self.fewshot_docs() is not None: ^^^^^^^^^^^^^^^^^^^ File "/home/lm-evaluation-harness/lm_eval/api/task.py", line 924, in fewshot_docs return self.dataset[self.config.fewshot_split] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/anaconda3/envs/llama/lib/python3.11/site-packages/datasets/dataset_dict.py", line 75, in **getitem** return super().**getitem**(k) ^^^^^^^^^^^^^^^^^^^^^^ KeyError: 'dev'
> 
> I found a solution to the issue of not being able to connect to huggingface.co without needing to download the dataset locally. You just need to modify the corresponding _URL link in the dataset's Python script to a mirror site. For example, if the error message is:
> 
> ```
> File "YourPath/mmlu_no_train.py", line 129, in _split_generators
>     archive = dl_manager.download(_URL)
> ```
> 
> Then reach to mmlu_no_train.py, and change
> 
> ```
> _URL = "https://huggingface.co/datasets/cais/mmlu/resolve/main/data.tar"
> ```
> 
> to
> 
> ```
> _URL = "https://hf-mirror.com/datasets/cais/mmlu/resolve/main/data.tar"
> ```
> 
> The same applies to other datasets.


  
This really helps me out!!! Many thanks!