Title: Loading of datasets changed in #3110 returns no examples 
Author: eladsegal (CONTRIBUTOR)
State: closed
Created: 2021-11-08T23:29:46Z
Labels: bug

## Describe the bug
Loading of datasets changed in https://github.com/huggingface/datasets/pull/3110 returns no examples:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
})
```

## Steps to reproduce the bug
Load any of the datasets that were changed in https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper")

# The problem only started with the commit of #3110
load_dataset("qasper", revision="b6469baa22c174b3906c631802a7016fedea6780")
```

## Expected results
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
```
Which can be received when specifying revision of the commit before https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper", revision="acfe2abda1ca79f0ce5c1896aa83b4b78af76b7d")
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.2.dev0 (master)
- Python version: 3.8.10
- PyArrow version: 3.0.0

Comments:
Hi @eladsegal, thanks for reporting.

I am sorry, but I can't reproduce the bug:
```
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("qasper")
Downloading: 5.11kB [00:00, ?B/s]
Downloading and preparing dataset qasper/qasper (download: 9.88 MiB, generated: 35.11 MiB, post-processed: Unknown size, total: 44.99 MiB) to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8...
Dataset qasper downloaded and prepared to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<?, ?it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
``` 

This makes me suspect that the origin of the problem might be the cache: I didn't have this dataset in my cache, although I guess you already had it, before the code change introduced by #3110.

@lhoestq might it be possible that the code change introduced by #3110 makes "inaccessible" all previously cached TAR-based datasets?
- Before the caching system downloaded and extracted the tar dataset
- Now it only downloads the tar dataset (no extraction is done)

I can't reproduce either in my environment (macos, python 3.7).

In your case it generates zero examples. This can only happen if the extraction of the TAR archive doesn't output the right filenames. Indeed if the `qasper` script can't find the right file to load, it's currently ignored and it returns zero examples. This case was not even considered when #3110 was developed since we considered the file names to be deterministic - and not depend on your environment.

Therefore here is my hypothesis:
- either the cache is corrupted somehow with an empty TAR archive
- OR I suspect that the issue comes from python 3.8


I just tried again on python 3.8 and I was able to reproduce the issue. Let me work on a fix

Ok I found the issue. It's not related to python 3.8 in itself though. This issue happens because your local installation of `datasets` is outdated compared to the changes to datasets in #3110

To fix this you just have to pull the latest changes from `master` :)

Let me know if that helps !

--------------

Here are more details about my investigation:

It's possible to reproduce this issue if you use `datasets<=1.15.1` or before b6469baa22c174b3906c631802a7016fedea6780 and if you load the dataset after revision b6469baa22c174b3906c631802a7016fedea6780. This is because `dl_manager.iter_archive` had issues at that time (and it was not used anywhere anyway).

In particular it was returning the absolute path to extracted files instead of the relative path of the file inside the archive. This was an issue because `dl_manager.iter_archive` isn't supposed to extract the TAR archive. Instead, it iterates over all the files inside the archive, without creating a directory with the extracted content.

Therefore if you want to use the datasets on `master`, make sure that you have an up-to-date local installation of `datasets` as well, or you may face incompatibilities like this.

Thanks!
But what about code that is already using older version of datasets? 
The reason I encountered this issue was that suddenly one of my repos with version 1.12.1 started getting 0 examples.
I handled it by adding `revision` to `load_dataset`, but I guess it would still be an issue for other users who doesn't know this.

Hi, in 1.12.1 it uses the dataset scripts from that time, not the one on master.

It only uses the datasets from master if you installed `datasets` from source, or if the dataset isn't available in your local version (in this case it shows a warning and it loads from master).


OK, I understand the issue a bit better now.
I see I wasn't on 1.12.1, but on 1.12.1.dev0 and since it is a dev version it uses master.
So users that use an old dev version must specify revision or else they'll encounter this problem.

BTW, when I opened the issue I installed the latest master version with
```
pip install git+git://github.com/huggingface/datasets@master#egg=datasets
```
and also used `download_mode="force_redownload"`, and it still returned 0 examples.
Now I deleted all of the cache and ran the code again, and it worked.
I'm not sure what exactly happened here, but looks like it was due to a mix of an unofficial version and its cache.

Thanks again!