Error when downloading a large dataset on slow connection.

Hi ! Is this an issue you have with `openwebtext` specifically or also with other datasets ?

It looks like the downloaded file is corrupted and can't be extracted using `tarfile`.
Could you try loading it again with
```python
import datasets
datasets.load_dataset("openwebtext", download_mode="force_redownload")
```

I receive the following error after about an hour trying to download the `openwebtext` dataset.

The code used is:
```python
import datasets
datasets.load_dataset("openwebtext")
```

> Traceback (most recent call last):                                                                                [4/28]
>   File "<stdin>", line 1, in <module>
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/load.py", line 610, in load_dataset
>     ignore_verifications=ignore_verifications,
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/builder.py", line 515, in download_and_prepare
>     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/builder.py", line 570, in _download_and_prepare
>     split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
>   File "/home/lucadiliello/.cache/huggingface/modules/datasets_modules/datasets/openwebtext/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02/openwebtext.py", line 62, in _split_generators
>     dl_dir = dl_manager.download_and_extract(_URL)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
>     return self.extract(self.download(url_or_urls))
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 235, in extract
>     num_proc=num_proc,
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
>     return function(data_struct)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
>     tar_file.extractall(output_path_extracted)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2000, in extractall
>     numeric_owner=numeric_owner)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2042, in extract
>     numeric_owner=numeric_owner)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2112, in _extract_member
>     self.makefile(tarinfo, targetpath)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 2161, in makefile
>     copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/tarfile.py", line 253, in copyfileobj
>     buf = src.read(remainder)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/lzma.py", line 200, in read
>     return self._buffer.read(size)
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/_compression.py", line 68, in readinto
>     data = self.read(len(byte_view))
>   File "/home/lucadiliello/anaconda3/envs/nlp/lib/python3.7/_compression.py", line 99, in read
>     raise EOFError("Compressed file ended before the "
> EOFError: Compressed file ended before the end-of-stream marker was reached


Add information about caching and verifications in "Load a Dataset" docs

documentation

Improvements or additions to documentation

Related to #215.

Missing improvements from @lhoestq's #1703.


Update XSUM Factuality DatasetCard

Update XSUM Factuality DatasetCard


Improvements regarding caching and fingerprinting

I few comments here for discussion:
- I'm not convinced yet the end user should really have to understand the difference between "caching" and 'fingerprinting", what do you think? I think fingerprinting should probably stay as an internal thing. Is there a case where we want cahing without fingerprinting or vice-versa?
- while I think the random fingerprint mechanism is smart, I have one question: when we disable caching or fingerprinting we also probably don't want the disk usage to grow so we should then try to keep only one cache file. Is it the case currently?
- the warning should be emitted only once per session if possible (we have a mechanism to do that in transformers, you should ask Lysandre/Sylvain)

About your points:
- Yes I agree, I just wanted to bring the discussion on this point. Until now fingerprinting hasn't been blocking for user experience. I'll probably remove the enable/disable fingerprinting function to keep things simple from the user's perspective.
- Right now every time a not in-place transform (i.e. map, filter) is applied, a new cache file is created. It is the case even if caching is disabled since disabling it only means that the cache file won't be reloaded. Therefore you're right that it might end up filling the disk with files that won't be reused. I like the idea of keeping only one cache file. Currently all the cache files are kept on disk until the user clears the cache. To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
```python
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(...)
```
In python it doesn't seem trivial to detect such changes. One thing that we can actually do on the other hand is store the cache files in a temporary directory that is cleared when the session closes. I think that's a good a simple solution for this problem.
- Yes good idea ! I don't like spam either :)

> * To be able to keep only one, we need to know if a dataset that has been transformed is still loaded or not. For example
>
> ```python
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(...)
> ```

I see what you mean. It's a tricky question. One option would be that if caching is deactivated we have a single memory mapped file and have copy act as a copy by reference instead of a copy by value. We will then probably want a `copy()` or `deepcopy()` functionality. Maybe we should think a little bit about it though.

- I like the idea of using a temporary directory per session!
- If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.
- So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?

> * If the default behavior when caching is disabled is to re-use the same file, I'm a little worried about people making mistakes and having to re-download and process from scratch.

We should distinguish between the caching from load_dataset (base dataset cache files) and the caching after dataset transforms such as map or filter (transformed dataset cache files). When disabling caching only the second type (for map and filter) doesn't reload from cache files.
Therefore nothing is re-downloaded. To re-download the dataset entirely the argument `download_mode="force_redownload"` must be used in `load_dataset`.
Do we have to think more about the naming to make things less confusing in your opinion ?

> * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?

There's no such `in_place` parameter in map, what do you mean exactly ?

I updated the PR:
- I removed the enable/disable fingerprinting function
- if caching is disabled arrow files are written in a temporary directory that is deleted when session closes
- the warning that is showed when hashing a transform fails is only showed once
- I added the `set_caching_enabled` function to the docs and explained the caching mechanism and its relation with fingerprinting

I would love to have some feedback :)

> > * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?
>
> There's no such `in_place` parameter in map, what do you mean exactly ?

Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:

```
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(..., in_place=True)
```

Case 1 returns a new reference using the new cache file, case 2 returns the same reference

> Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:
>
> ```
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(..., in_place=True)
> ```
>
> Case 1 returns a new reference using the new cache file, case 2 returns the same reference

Ok I see !
`in_place` is a parameter that is used in general to designate a transform so I would name that differently (maybe `overwrite` or something like that).
Not sure if it's possible to update an already existing arrow file that is memory-mapped, let me check real quick.
Also it's possible to call `dataset2.cleanup_cache_files()` to delete the other cache files if we create a new one after the transform. Or even to get the cache file with `dataset1.cache_files` and let the user remove them by hand.

EDIT: updating an arrow file in place is not part of the current API of pyarrow, so we would have to make new files.

This PR adds these features:
- Enable/disable caching
    If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    It is equivalent to setting `load_from_cache` to `False` in dataset transforms.
```python
from datasets import set_caching_enabled

set_caching_enabled(False)
```
- Allow unpicklable functions in `map`
    If an unpicklable function is used, then it's not possible to hash it to update the dataset fingerprint that is used to name cache files. To workaround that, a random fingerprint is generated instead and a warning is raised.
```python
logger.warning(
    f"Transform {transform} couldn't be hashed properly, a random hash was used instead. "
    "Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. "
    "If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything."
)
```

and also (open to discussion, EDIT: actually NOT included):
- Enable/disable fingerprinting
    Fingerprinting allows to have one deterministic fingerprint per dataset state.
    A dataset fingerprint is updated after each transform.
    Re-running the same transforms on a dataset in a different session results in the same fingerprint.
    Disabling the fingerprinting mechanism makes all the fingerprints random.
    Since the caching mechanism uses fingerprints to name the cache files, then cache file names will be different.
    Therefore disabling fingerprinting will prevent the caching mechanism from reloading datasets files that have already been computed.
    Disabling fingerprinting may speed up the lib for users that don't care about this feature and don't want to use caching.
```python
from datasets import set_fingerprinting_enabled

set_fingerprinting_enabled(False)
```

Other details:
- I renamed the `fingerprint` decorator to `fingerprint_transform` since the name was clearly not explicit. This decorator is used on dataset transform functions to allow them to update fingerprints.
- I added some `ignore_kwargs` when decorating transforms with `fingerprint_transform`, to make the fingerprint update not sensible to kwargs like `load_from_cache` or `cache_file_name`.

Todo: tests for set_fingerprinting_enabled + documentation for all the above features


Fix importlib metdata import in py38

In Python 3.8 there's no need to install `importlib_metadata` since it already exists as `importlib.metadata` in the standard lib.


Some datasets miss dataset_infos.json or dummy_data.zip

Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json :

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip :

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .


Update Curiosity dialogs DatasetCard

Update Curiosity dialogs DatasetCard

There are some entries in the data fields section yet to be filled. There is little information regarding those fields.


Update DBRD dataset card and download URL

not sure why the CI was not triggered though

I've added the Dutch Bood Review Dataset (DBRD) during the recent sprint. This pull request makes two minor changes:

1.  I'm changing the download URL from Google Drive to the dataset's GitHub release package. This is now possible because of PR #1316.
2. I've updated the dataset card.

Cheers! 😄


Update Coached Conv Pref DatasetCard

Really cool!

Can you add some task tags for `dialogue-modeling` (under `sequence-modeling`) and `parsing` (under `structured-prediction`)?

Update Coached Conversation Preferance DatasetCard


Update DialogRE DatasetCard

Same as #1698, can you add a task tag for dialogue-modeling (under sequence-modeling) :) ?

Update the information in the dataset card for the Dialog RE dataset.