Title: Load local dataset error
Author: TXacs (NONE)
State: open
Created: 2022-03-18T03:32:49Z
Labels: bug

dataset bug

When i used the datasets==1.11.0， it's all right. Util update the latest version, it get the error like this:
```
>>> from datasets import load_dataset
>>> data_files={'train': ['/ssd/datasets/imagenet/pytorch/train'], 'validation': ['/ssd/datasets/imagenet/pytorch/val']}
>>> ds = load_dataset('nateraw/image-folder', data_files=data_files, cache_dir='./', task='image-classification')
[] https://huggingface.co/datasets/nateraw/image-folder/resolve/main/ /dat/txacs/git/txacs/examples/image-classification/https:/huggingface.co/datasets/nateraw/image-folder/resolve/main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/load.py", line 1671, in load_dataset
    **config_kwargs,
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/load.py", line 1521, in load_dataset_builder
    **config_kwargs,
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/builder.py", line 1031, in __init__
    super().__init__(*args, **kwargs)
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/builder.py", line 255, in __init__
    sanitize_patterns(data_files), base_path=base_path, use_auth_token=use_auth_token
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/data_files.py", line 584, in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/data_files.py", line 546, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/data_files.py", line 196, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/gf3/home/txacs/gv3/anaconda3/envs/txacs/lib/python3.6/site-packages/datasets/data_files.py", line 146, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/ssd/datasets/imagenet/pytorch/train' at /dat/txacs/git/txacs/examples/image-classification/https:/huggingface.co/datasets/nateraw/image-folder/resolve/main
```
I need some help to solve the problem, thanks!

Comments:
Hi! Instead of @nateraw's `image-folder`, I suggest using the newly released `imagefolder` dataset:
```python
>>> from datasets import load_dataset
>>> data_files={'train': ['/ssd/datasets/imagenet/pytorch/train/**'], 'validation': ['/ssd/datasets/imagenet/pytorch/val/**']}
>>> ds = load_dataset('imagefolder', data_files=data_files, cache_dir='./', task='image-classification')
```


Let us know if that resolves the issue.

> Hi! Instead of @nateraw's `image-folder`, I suggest using the newly released `imagefolder` dataset:
> 
> ```python
> >>> from datasets import load_dataset
> >>> data_files={'train': ['/ssd/datasets/imagenet/pytorch/train/**'], 'validation': ['/ssd/datasets/imagenet/pytorch/val/**']}
> >>> ds = load_dataset('imagefolder', data_files=data_files, cache_dir='./', task='image-classification')
> ```
> 
> Let us know if that resolves the issue.

Sorry, replied late.
Thanks a lot! It's worked for me. But it seems much slower than before, and now gets stuck.....

```
>>> from datasets import load_dataset
>>> data_files={'train': ['/ssd/datasets/imagenet/pytorch/train/**'], 'validation': ['/ssd/datasets/imagenet/pytorch/val/**']}
>>> ds = load_dataset('imagefolder', data_files=data_files, cache_dir='./', task='image-classification')
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1281167/1281167 [00:02<00:00, 437283.97it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50001/50001 [00:00<00:00, 89094.29it/s]
Using custom data configuration default-baebca6347576b33
Downloading and preparing dataset image_folder/default to ./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091...
Downloading data files #0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 82289.56obj/s]
Downloading data files #1: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:01<00:00, 73559.11obj/s]
Downloading data files #2: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 81600.46obj/s]
Downloading data files #3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:01<00:00, 79691.56obj/s]
Downloading data files #4: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 82341.37obj/s]
Downloading data files #5: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:01<00:00, 75784.46obj/s]
Downloading data files #6: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 81466.18obj/s]
Downloading data files #7: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 82320.27obj/s]
Downloading data files #8: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:01<00:00, 78094.00obj/s]
Downloading data files #9: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 84057.59obj/s]
Downloading data files #10: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 83082.31obj/s]
Downloading data files #11: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:01<00:00, 79944.21obj/s]
Downloading data files #12: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 84569.77obj/s]
Downloading data files #13: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 84949.63obj/s]
Downloading data files #14: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80073/80073 [00:00<00:00, 80666.53obj/s]
Downloading data files #15: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80072/80072 [00:01<00:00, 76723.20obj/s]
^[[Bloading data files #8:  94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌       | 75061/80073 [00:00<00:00, 82609.89obj/s]
Downloading data files #9:  85%|████████████████████████████████████████████████████████████████████████████████████████████████████▍                 | 68120/80073 [00:00<00:00, 83868.54obj/s]
Downloading data files #9:  96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏    | 76784/80073 [00:00<00:00, 84722.34obj/s]
Downloading data files #10:  75%|███████████████████████████████████████████████████████████████████████████████████████▋                             | 59995/80073 [00:00<00:00, 84148.19obj/s]
Downloading data files #10:  97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████    | 77412/80073 [00:00<00:00, 85724.53obj/s]
Downloading data files #11:  71%|███████████████████████████████████████████████████████████████████████████████████▎                                 | 57032/80073 [00:00<00:00, 79930.58obj/s]
Downloading data files #11:  92%|███████████████████████████████████████████████████████████████████████████████████████████████████████████          | 73277/80073 [00:00<00:00, 78091.27obj/s]
Downloading data files #12:  86%|█████████████████████████████████████████████████████████████████████████████████████████████████████                | 69125/80073 [00:00<00:00, 84723.02obj/s]
Downloading data files #12:  97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋   | 77803/80073 [00:00<00:00, 85351.59obj/s]
Downloading data files #13:  75%|████████████████████████████████████████████████████████████████████████████████████████▏                            | 60356/80073 [00:00<00:00, 84833.35obj/s]
Downloading data files #13:  97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████    | 77368/80073 [00:00<00:00, 84475.10obj/s]
Downloading data files #14:  72%|████████████████████████████████████████████████████████████████████████████████████▍                                | 57751/80073 [00:00<00:00, 80727.33obj/s]
Downloading data files #14:  92%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▏        | 74022/80073 [00:00<00:00, 78703.16obj/s]
Downloading data files #15:  78%|███████████████████████████████████████████████████████████████████████████████████████████▋                         | 62724/80072 [00:00<00:00, 78387.33obj/s]
Downloading data files #15:  99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 78933/80072 [00:01<00:00, 79353.63obj/s]
```

Wait a long time, it completed. I don't know why it's so slow...

You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.

> You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.

Thanks！It's worked well.

> You can pass `ignore_verifications=True` in `load_dataset` to make it fast (to skip checksum verification). I'll add this tip to the docs.

I find current `load_dataset` loads ImageNet still slowly, even add `ignore_verifications=True`.
First loading, it costs about 20 min in my servers.
```
real	19m23.023s
user	21m18.360s
sys	7m59.080s
```

Second reusing, it costs about 15 min in my servers.
```
real	15m20.735s
user	12m22.979s
sys	5m46.960s
```

I think it's too much slow, is there other method to make it faster?

And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
```python
def collate_fn(examples):
    pixel_values = torch.stack([example["pixel_values"] for example in examples])
    labels = torch.tensor([example["labels"] for example in examples])
    return {"pixel_values": pixel_values, "labels": labels}
```
How to know the keys of example?

Loading the image files slowly, is it because the multiple processes load files at the same time?

Could you please share the output you get after the second loading? Also, feel free to interrupt (`KeyboardInterrupt`) the process while waiting for it to end and share a traceback to show us where the process hangs.   

> And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
> 
> ```python
> def collate_fn(examples):
>     pixel_values = torch.stack([example["pixel_values"] for example in examples])
>     labels = torch.tensor([example["labels"] for example in examples])
>     return {"pixel_values": pixel_values, "labels": labels}
> ```
> 
> How to know the keys of example?

What do you mean by "could you make some changes".The `ViT` script doesn't remove unused columns by default, so the keys of an example are equal to the columns of the given dataset.



> Could you please share the output you get after the second loading? Also, feel free to interrupt (`KeyboardInterrupt`) the process while waiting for it to end and share a traceback to show us where the process hangs.
> 
> > And in transformers the [ViT example](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-classification/run_image_classification.py), could you make some changes ? Like the `collect_fn`
> > ```python
> > def collate_fn(examples):
> >     pixel_values = torch.stack([example["pixel_values"] for example in examples])
> >     labels = torch.tensor([example["labels"] for example in examples])
> >     return {"pixel_values": pixel_values, "labels": labels}
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > How to know the keys of example?
> 
> What do you mean by "could you make some changes".The `ViT` script doesn't remove unused columns by default, so the keys of an example are equal to the columns of the given dataset.

Thanks for your reply!

1. I did not record the second output, so I run it again.  
```
(merak) txacs@master:/dat/txacs/test$ time python test.py 
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1281167/1281167 [00:02<00:00, 469497.89it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50001/50001 [00:00<00:00, 70123.73it/s]
Using custom data configuration default-baebca6347576b33
Reusing dataset image_folder (./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00,  5.37s/it]
Loading cached processed dataset at ./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091/cache-cd3fbdc025e03f8c.arrow
Loading cached processed dataset at ./image_folder/default-baebca6347576b33/0.0.0/ee92df8e96c6907f3c851a987be3fd03d4b93b247e727b69a8e23ac94392a091/cache-b5a9de701bbdbb2b.arrow
DatasetDict({
    train: Dataset({
        features: ['image', 'labels'],
        num_rows: 1281167
    })
    validation: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
})

real	10m10.413s
user	9m33.195s
sys	2m47.528s
```
Although it cost less time than the last, but still slowly.

2. Sorry, forgive my poor statement. I solved it, updating to new script 'run_image_classification.py'.

Thanks for rerunning the code to record the output. Is it the `"Resolving data files"` part on your machine that takes a long time to complete, or is it `"Loading cached processed dataset at ..."˙`? We plan to speed up the latter by splitting bigger Arrow files into smaller ones, but your dataset doesn't seem that big, so not sure if that's the issue.

> Thanks for rerunning the code to record the output. Is it the `"Resolving data files"` part on your machine that takes a long time to complete, or is it `"Loading cached processed dataset at ..."˙`? We plan to speed up the latter by splitting bigger Arrow files into smaller ones, but your dataset doesn't seem that big, so not sure if that's the issue.

Sounds good! The main position, which costs long time, is from program starting to `"Resolving data files"`. I hope you can solve it early, thanks!

I'm getting this problem. Script has been stuck at this part for the past 15 or so minutes:
 
`Resolving data files: 100%|█████████████████████████████████████████| 107/107 [00:00<00:00, 472.74it/s]`

I had everything working fine on an AWS EC2 node with a single GPU. Then I created an image based on the single GPU machine, and spun up a new one with 4 GPUs, so I got all of the training data ready at .cache. 

Turned off all checks with `verification_mode='no_checks'`. Logged in with huggingface-cli again just to be sure.

Interrupting shows the code is stuck here:

```
File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 200, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 336, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/arrow_reader.py", line 357, in read_table
    return table_cls.from_file(filename)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/table.py", line 1059, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/datasets/table.py", line 66, in _memory_mapped_arrow_table_from_file
    pa_table = opened_stream.read_all()
```

Is it just going to take a while or am I going to run out of money? :sweat_smile: 

edit: ping @mariosasko