Repository: EleutherAI/lm-evaluation-harness
Issue: Local dataset or model path support
Root cause: Documentation Deficiency

Hi~I would like to use dataset or model locally, but I have not found the way by configuring tasks/model. Is their any way to use model or dataset locally through appropriate settings or will this function be supported?

Referring to some issues mentioned vefore, I have tried passing local model path on --pretrained model args, it works. However, when I tried use the same way on --task args, I failed. Referring to the new task guide doc and https://github.com/EleutherAI/lm-evaluation-harness/issues/547, my command is as follow:
lm_eval --model hf     --model_args pretrained=/home/work/local-model-ds-harness-test/local-model/     --tasks /home/work/local-model-ds-harness-test/local-dataset/local-hellaswag-task/syc-hellaswag2     --device cuda:0     --batch_size 8
The task path is a duplicate of dataset hellaswag, my config file is as follow
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/42312157/16b3ace7-8567-4f34-b56a-9be52b6d1a85)
Contents under my datadir is hellaswag datasets, I have used load_dataset and save_to_path, the structure is as follow:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/42312157/9f836d27-6146-4fdd-8546-3896fae22f0e)
The error is as follow
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/42312157/0aa09d68-2238-4d54-bddd-9f5a5a3a78cc)
So, I want to know, is there any way that I could use dataset locally. Is the way I use it wrong? Sincerely appreciation on your help~



Hi! you can simply use `--tasks syc-hellaswag2` (the `task` field) now, after modifying the `dataset_path` on the task config like you did here. If this config is in another folder other than the default one you can also use `--include_path DIR` to also look in that directory. Adding `--verbosity DEBUG` also prints out the intermediate outputs in case of any problems in loading.

> Hi! you can simply use `--tasks syc-hellaswag2` (the `task` field) now, after modifying the `dataset_path` on the task config like you did here. If this config is in another folder other than the default one you can also use `--include_path DIR` to also look in that directory. Adding `--verbosity DEBUG` also prints out the intermediate outputs in case of any problems in loading.

It does not work. I have tried to add it in lm_eval/tasks dir, but it told me that ValueError: `Tasks not found: syc-hellaswag2`. I am sure if I use my dataset name on the huggingface it works, so I think such task is not added successfully after I change dataset_path to local path. The result is as follow,
<img width="1424" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/42312157/4a8eb0e1-6c8c-47c2-a642-101ff007ef68">


Internally we use the datasets API by calling [datasets.load_dataset](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset) (`dataset_path`, `dataset_name` etc.). See if it loads through that to narrow down the potential issue. You can also pass in other args using
```
dataset_kwargs:
  data_files: /path/to/my/json
```
if needs be.

> Internally we use the datasets API by calling [datasets.load_dataset](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset) (`dataset_path`, `dataset_name` etc.). See if it loads through that to narrow down the potential issue. You can also pass in other args using
> 
> ```
> dataset_kwargs:
>   data_files: /path/to/my/json
> ```
> 
> if needs be.

So I think I should change dataset_path to arrow or json? But The same issue occurs when I add this and change the dataset_path param in the task yaml

I tried:

```python
import datasets
x = datasets.load_dataset("hellaswag")
x.save_to_disk("hellaswag_local/")

y = datasets.load_dataset(path="hellaswag", data_dir="hellaswag_local/")
```
which saved the configs of the train, val, and test splits in their own sub-folders and loaded them back successfully. In the yaml, I just copied the args I used to load the dataset from the local folder (this might be different if you modified the dataset):

```yaml
dataset_path: hellaswag
dataset_kwargs:
    data_dir: hellaswag_local/
```


@lintangsutawika did #1127 break our recommendations for local datasets? Looks like there is a conflict here where we search for a dataset loading python script but can't find it, despite an HF datasets - compliant dir at that location.

This isn't a bug per se, as I think we do want to support custom dataset scripts at the moment if a user desires it, but we should make sure to document using `save_to_disk()` / `save_to_path()` to run using a saved HF dataset so that this doesn't cause confusion.

Running locally / offline might deserve its own docs subsection / FAQ.


This issue seems to be a different use-case where the local dataset is a downloaded version of the a hf dataset? If there isn't any difference than maybe setting up HF's CACHE_DIR os variable would help locate the data.

#1127 is meant for tasks where users want to supply their own python script (which HF datasets all have to download and prepare the final datafiles)

Otherwise for local data like jsonl or csv.
```
dataset_path: "json"
dataset_name: null
validation_split: train
dataset_kwargs:
  data_files: /path/to/json
```

> This issue seems to be a different use-case where the local dataset is a downloaded version of the a hf dataset? If there isn't any difference than maybe setting up HF's CACHE_DIR os variable would help locate the data.
> 
> #1127 is meant for tasks where users want to supply their own python script (which HF datasets all have to download and prepare the final datafiles)

I have changed the config file, setting the dataset_path to my own duplicate dataset path `ycsong-eugene/syc-hellaswag2` and the datadir to my local path, which uses the the same function to save as you mentioned, however, it tells me the error as follow, 
<img width="1437" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/42312157/c21f0541-1e76-4937-b5f9-cc22b253d728">
but I am sure it is the right place to save the data~

> @lintangsutawika did #1127 break our recommendations for local datasets? Looks like there is a conflict here where we search for a dataset loading python script but can't find it, despite an HF datasets - compliant dir at that location.
> 
> This isn't a bug per se, as I think we do want to support custom dataset scripts at the moment if a user desires it, but we should make sure to document using `save_to_disk()` / `save_to_path()` to run using a saved HF dataset so that this doesn't cause confusion.
> 
> Running locally / offline might deserve its own docs subsection / FAQ.

Hi~ thanks for your concern. My aim is supporting local dataset , downloaded from huggingface. So, is it supported right now? I have searched the doc carefully but do not find what I want. Will related doc be supported in the future? I am looking fooking forward to your reply sincerely~

You might need to follow the documentation from `datasets`
https://huggingface.co/docs/datasets/loading#local-loading-script

`dataset_path` and `dataset_name` maps to the args `path` and `name` in `load_dataset`
https://huggingface.co/docs/datasets/v2.15.0/en/package_reference/loading_methods#datasets.load_dataset

If you can make sure that you can load your local dataset using `load_dataset` that it should also work on lm-eval (this would be a goo way to test it)

> Hi~ thanks for your concern. My aim is supporting local dataset , downloaded from huggingface. So, is it supported right now? I have searched the doc carefully but do not find what I want. Will related doc be supported in the future? I am looking fooking forward to your reply sincerely~

We should be able to add this by mid-week next week and document it, if it is indeed a gap in our support!

> https://github.com/EleutherAI/lm-evaluation-harness/pull/1127 is meant for tasks where users want to supply their own python script (which HF datasets all have to download and prepare the final datafiles

HF is moving away from this towards parquets/arrow, though it will still be supported.


> Hi~ thanks for your concern. My aim is supporting local dataset , downloaded from huggingface. So, is it supported right now? I have searched the doc carefully but do not find what I want.

Ok this is definitely working (with dataset saved with `save_to_disk()`:

```yaml
dataset_path: arrow
dataset_kwargs:
  data_files:
    train: hellaswag_local/train/data-00000-of-00001.arrow
    validation: hellaswag_local/validation/data-00000-of-00001.arrow
```

> > Hi~ thanks for your concern. My aim is supporting local dataset , downloaded from huggingface. So, is it supported right now? I have searched the doc carefully but do not find what I want.
> 
> Ok this is definitely working (with dataset saved with `save_to_disk()`:
> 
> ```yaml
> dataset_path: arrow
> dataset_kwargs:
>   data_files:
>     train: hellaswag_local/train/data-00000-of-00001.arrow
>     validation: hellaswag_local/validation/data-00000-of-00001.arrow
> ```

Thanks a lot~ I have confirmed this way works~ thank you very much~

> > Hi~ thanks for your concern. My aim is supporting local dataset , downloaded from huggingface. So, is it supported right now? I have searched the doc carefully but do not find what I want. Will related doc be supported in the future? I am looking fooking forward to your reply sincerely~
> 
> We should be able to add this by mid-week next week and document it, if it is indeed a gap in our support!
> 
> > #1127 is meant for tasks where users want to supply their own python script (which HF datasets all have to download and prepare the final datafiles
> 
> HF is moving away from this towards parquets/arrow, though it will still be supported.

Thanks a lot, I will keep paying attention on it and please sync this update here~

Closing in favor of #1233 since we do currently support this, just want to document it more explicitly w.r.t. `save_to_disk()`.