Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to specify dataset paths？
Root cause: Documentation Deficiency


Hi, can you provide more details about your issue?

If you mean dataset paths for huggingface dataset, you can go and change it in the yaml file of each task.

Thanks. But if I want to use local datasets, how should I change the command and how to organize the datasets' structure?

I think so long you arrange your local dataset in the huggingface format and setting dataset_path to point to your dataset path, it should work fine.

Where is yaml?

I just run `python llms_eval.py --model llava   --model_args pretrained="./checkpoints/llava-qwen-4b-finetune-490/checkpoint-1500/,conv_template=qwen"   --tasks mme  --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/ `

I think it would be better to specific a root local dataset path? (we might download dataset to eval which needed )

There are many servers hard to connect internet but only share disks with develop machine, load from hf shouldn't be a force choice

@lucasjinreal , the yaml file is in the folder of each task. For example `lmms_eval/tasks/ai2d/ai2d.yaml`

If you don't want to change the dataset_path one by one, you can download all the dataset first outside the server using the command `lmms_eval --tasks list_with_num` and then uploading the cache folder to your server disk. You can then export `HF_HOME` to your server cache folder and reuse cache dataset each time.

@kcz358  Hi, is possible to send a dataset_root from args command line? So that it could be not such cubersome.

You can set the environment variable to a folder and make your offline datasets all in this folder.

It will be something like:

```
export HF_HOME=/user/boli01/workspace/.cache/huggingface
```

And your offline dataset be like 

```
/user/boli01/workspace/.cache/huggingface/dataset_A
```


How can I offline download the dataset using git clone directly to cache folder?

It looks like it can not found it properly

I still get this error:

huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.

I can not create any tokens on servers machine.

It still could be better provide a dataset_root path from main, so that users can download by their own.

@lucasjinreal May I asked are you now encountering this error during evaluation or during download dataset.

By far our suggestion is to download the dataset you need in a machine that can access to huggingface and transfer the dataset to your server.

If you want to specify a dataset root from you local file, you can specify it by `export HF_HOME=<your dataset dir>`

You can also try setting token=False in the task folder. As long as you can make sure your local dataset path can be loaded by using `load_dataset` everything should be fine.

@kcz358 Hi, I set the path to my local, and downloaded for example MME data to that folder, but the eval doesn't read from it at all.

I have already set variables. 

How to set token false? I didn't see a key in yaml.


Still, I would recommend if task path can set directly from local and read from it if it available rather than ask to download even if it there.

@lucasjinreal , You should make sure your path can be loaded through `load_dataset` method 

In your yaml file, you can set
```
dataset_kwargs:
  token: False
```

And for your last recommendation, we have already suggested multiple times that you can control your cache folder path by `export HF_HOME=<your cache folder>`. You just need to download the data using hf load_dataset outside and upload to your server. HF will manage the dataset structure for you and you don't need to change it. Using the command `lmms_eval --tasks list_with_num` will download every dataset in our repo for you in the cache folder of your external server and you just need to upload the cache folder to your server.

And again, if you want to change your dataset path, you should check whether the path can be loaded by `load_dataset` method

Hi, I can't load the dataset on an offline machine.

```
import os
os.environ['HF_HOME'] = "/share_data/hf_home"
os.environ['HF_DATASETS_OFFLINE'] = '1'

import datasets
from datasets import DownloadConfig, Image, Sequence

dataset_kwargs = {
    "token": "hf_token", # this need to be replaced with a user token
}

download_config = DownloadConfig(cache_dir=dataset_kwargs.get('cache_dir', None))
download_config.max_retries = dataset_kwargs.get("max_retries", 10) if dataset_kwargs is not None else 10
download_config.num_proc = dataset_kwargs.get("num_proc", 8) if dataset_kwargs is not None else 8
download_config.local_files_only = dataset_kwargs.get("local_files_only", False) if dataset_kwargs is not None else False

print(download_config)

DATASET_PATH="lmms-lab/MME"
DATASET_NAME=None
dataset = datasets.load_dataset(
    path=DATASET_PATH,
    name=DATASET_NAME,
    download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
    download_config=download_config,
    **dataset_kwargs if dataset_kwargs is not None else {},
)
```

On the online machine, I have download the dataset:
```ll /share_data/hf_home/datasets```
<img width="912" alt="image" src="https://github.com/user-attachments/assets/be3099c1-df87-4b67-8d4d-cc999bcd1522">


And I can load the dataset successfully:
<img width="1094" alt="image" src="https://github.com/user-attachments/assets/d6a6aba1-f087-468f-90bc-9bacaa8a9cb2">

However, on the offline machine, I can't load the dataset:
<img width="1173" alt="image" src="https://github.com/user-attachments/assets/7ae692ff-1f2d-4f2c-8dcb-15fe322b7818">

Do you have any advice? @Luodian @kcz358 

> Hi, I can't load the dataset on an offline machine.
> 
> ```
> import os
> os.environ['HF_HOME'] = "/share_data/hf_home"
> os.environ['HF_DATASETS_OFFLINE'] = '1'
> 
> import datasets
> from datasets import DownloadConfig, Image, Sequence
> 
> dataset_kwargs = {
>     "token": "hf_token", # this need to be replaced with a user token
> }
> 
> download_config = DownloadConfig(cache_dir=dataset_kwargs.get('cache_dir', None))
> download_config.max_retries = dataset_kwargs.get("max_retries", 10) if dataset_kwargs is not None else 10
> download_config.num_proc = dataset_kwargs.get("num_proc", 8) if dataset_kwargs is not None else 8
> download_config.local_files_only = dataset_kwargs.get("local_files_only", False) if dataset_kwargs is not None else False
> 
> print(download_config)
> 
> DATASET_PATH="lmms-lab/MME"
> DATASET_NAME=None
> dataset = datasets.load_dataset(
>     path=DATASET_PATH,
>     name=DATASET_NAME,
>     download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
>     download_config=download_config,
>     **dataset_kwargs if dataset_kwargs is not None else {},
> )
> ```
> 
> On the online machine, I have download the dataset: `ll /share_data/hf_home/datasets` <img alt="image" width="912" src="https://private-user-images.githubusercontent.com/73805072/370593019-be3099c1-df87-4b67-8d4d-cc999bcd1522.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcyNDg5MzIsIm5iZiI6MTcyNzI0ODYzMiwicGF0aCI6Ii83MzgwNTA3Mi8zNzA1OTMwMTktYmUzMDk5YzEtZGY4Ny00YjY3LThkNGQtY2M5OTliY2QxNTIyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI1VDA3MTcxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA2NDc1NGMwOTNjMzg3ODRhY2I1Y2IxNzgwNzUxMTcyZjk3NmFlMDdhMDc0ODQ3ZmY4MThlN2NkY2RlY2ViZmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.O_qqhuXSCspLL_qWA93JzCzkDM5baKS1owoZGsmzbmA">
> 
> And I can load the dataset successfully: <img alt="image" width="1094" src="https://private-user-images.githubusercontent.com/73805072/370592533-d6a6aba1-f087-468f-90bc-9bacaa8a9cb2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcyNDg5MzIsIm5iZiI6MTcyNzI0ODYzMiwicGF0aCI6Ii83MzgwNTA3Mi8zNzA1OTI1MzMtZDZhNmFiYTEtZjA4Ny00NjhmLTkwYmMtOWJhY2FhOGE5Y2IyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI1VDA3MTcxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA2MTE0YTcyYmI4NDg0MGI4NDFkZjc5YmZmNGNkYzZmNmFmZDAyOTU5MzYzNmVkZmZlNDEwODIzZDZmODRiZmYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.-jhDIwYNptl-fuKvOAdhX5F3uhk3sUboJdpOtOVZJao">
> 
> However, on the offline machine, I can't load the dataset: <img alt="image" width="1173" src="https://private-user-images.githubusercontent.com/73805072/370593466-7ae692ff-1f2d-4f2c-8dcb-15fe322b7818.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjcyNDg5MzIsIm5iZiI6MTcyNzI0ODYzMiwicGF0aCI6Ii83MzgwNTA3Mi8zNzA1OTM0NjYtN2FlNjkyZmYtMWYyZC00ZjJjLThkY2ItMTVmZTMyMmI3ODE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTI1VDA3MTcxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI3ZmIzMjY3OTdhMGQ0ZDc4OTc1OGRhNGQ1OWVlNDdmNWNkMTZiMWIxZjAxYmI4MDE4OTFhYTkyOWE3MzQxNzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Hzci55tmstDtVR0hRJclzFbFRs_CwD5UqP3QHDnYBcs">
> 
> Do you have any advice? @Luodian @kcz358

Solved after upgrading datasets to `2.20.0`
```
pip install datasets==2.20.0
```

https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/15#issuecomment-2373251290
datasets==2.20.0 didn't solve my problem...
Here's my script
```shell
# cd /path/to/lmms-eval
# python3 -m pip install -e .;

# cd /path/to/LLaVA-NeXT;
# python3 -m pip install -e ".[train]";
export HF_HOME="/path/to/my/hf_home"
export HF_DATASETS_OFFLINE=1
export HF_HUB_OFFLINE=1
export CUDA_VISIBLE_DEVICES=1,2,3,4

TASK=olympiadbench_test_en
CKPT_PATH=xxxxx
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 4 --main_process_port 12345 -m lmms_eval \
    --model qwen_vl \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
``` 
And my file system is like /path/to/my/hf_home/lmms-lab/OlympiadBench, where did I go wrong?

Could you try to set the yaml file to look like this?

```yaml
dataset_path: local_path
task: "ai2d"
dataset_kwargs:
  token: True
  load_from_disk: True
```

And set a breakpoint around these lines to check if the code reaches here.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/931d88d15a057dc574fc183200fbd3eaf10e0906/lmms_eval/api/task.py#L1039-L1043



> Could you try to set the yaml file to look like this?
> 
> dataset_path: local_path
> task: "ai2d"
> dataset_kwargs:
>   token: True
>   load_from_disk: True
> And set a breakpoint around these lines to check if the code reaches here.
> 
> [lmms-eval/lmms_eval/api/task.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/931d88d15a057dc574fc183200fbd3eaf10e0906/lmms_eval/api/task.py#L1039-L1043)
> 
> Lines 1039 to 1043 in [931d88d](/EvolvingLMMs-Lab/lmms-eval/commit/931d88d15a057dc574fc183200fbd3eaf10e0906)
> 
>  if dataset_kwargs is not None and "load_from_disk" in dataset_kwargs and dataset_kwargs["load_from_disk"]: 
>      dataset_kwargs.pop("load_from_disk") 
>      # using local task in offline environment, need to process the online dataset into local format via 
>      # `ds = load_datasets("lmms-lab/MMMU")` 
>      self.dataset = datasets.load_from_disk(path=self.DATASET_PATH, name=self.DATASET_NAME)

Thanks to your reply. I have already debugged it and found setting `dataset_path: abs_local_path` in yaml file would help. It seems that `export HF_HOME="/path/to/my/hf_home"` didn't work as expected.

> Could you try to set the yaml file to look like this?你能尝试将 yaml 文件设置为这样吗？
> 
> dataset_path: local_path
> task: "ai2d"
> dataset_kwargs:
>   token: True
>   load_from_disk: True
> And set a breakpoint around these lines to check if the code reaches here.并在这些行周围设置一个断点，以检查代码是否到达此处。
> 
> [lmms-eval/lmms_eval/api/task.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/931d88d15a057dc574fc183200fbd3eaf10e0906/lmms_eval/api/task.py#L1039-L1043)
> 
> Lines 1039 to 1043 in [931d88d](/EvolvingLMMs-Lab/lmms-eval/commit/931d88d15a057dc574fc183200fbd3eaf10e0906)
> 
>  if dataset_kwargs is not None and "load_from_disk" in dataset_kwargs and dataset_kwargs["load_from_disk"]: 
>      dataset_kwargs.pop("load_from_disk") 
>      # using local task in offline environment, need to process the online dataset into local format via 
>      # `ds = load_datasets("lmms-lab/MMMU")` 
>      self.dataset = datasets.load_from_disk(path=self.DATASET_PATH, name=self.DATASET_NAME)

This is such a clear and common requirement. Why haven't you added an argument for it directly?

Hi can you help to raise a PR to help us improve?