Title: Specify split post processing + Add post processing resources downloading
Author: lhoestq (MEMBER)
State: closed
Created: 2020-07-24T09:29:19Z

Previously if you tried to do

```python
from nlp import load_dataset
wiki = load_dataset("wiki_dpr", "psgs_w100_with_nq_embeddings", split="train[:100]", with_index=True)
```
Then you'd get an error `Index size should match Dataset size...`
This was because it was trying to use the full index (21M elements).

To fix that I made it so post processing resources can be named according to the split.

I'm going to add tests on post processing too.

Note that the CI will fail as I added a new argument in `_post_processing_resources`: the AWS version of wiki_dpr fails, and there's also an error telling that it is not synced (it'll be synced once it's merged):
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_wiki_dpr
FAILED tests/test_hf_gcp.py::TestDatasetSynced::test_script_synced_with_s3_wiki_dpr
```

EDIT: I did a change to ignore the script hash to locate the arrow files on GCS, so I removed the sync test. It was there just because of the hash logic for files on GCS

Comments:
I was using a hack in `wiki_dpr` to download the index from GCS even for the configurations without the embeddings.
However as GCS is something internal, I changed the logic to add a download step for indexes directly in the dataset script, using the `DownloadManager`.

This change was directly linked to the changes I did to take into account the split name in the post processing, so I included this change in this PR too.

To summarize:

Dataset builders can now implement
- `_post_processing_resources(split)`: return a dict `resource_name -> resource_file_name`. It defines the additional resources such as indexes or arrow files that you need in post processing
- `_download_post_processing_resources(split, resource_name, dl_manager))`: if some resources can be downloaded, you can use the download_manager to download them
- `_post_process(dataset, resources_path)`: (main function for post processing) given a dataset, you can apply dataset transforms or add indexes. For resources that have been downloaded, you can load them. For the others, you can generate and save them. The paths to load/save resources are in `resources_path` which is a dictionary `resource_name -> resource_path`

About the CI:
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_wiki_dpr
```
It fails because I changed the input of post processing functions (to include the split name)

I started to add metadata in the DatasetInfo.
Note that because there are new fields, **ALL the dataset_info[s].json generated after these changes won't be loadable from older versions of the lib**

Right now it looks like this:
```json
        "post_processing_resources_checksums": {
            "train": {
                "embeddings_index": {
                    "num_bytes": 30720045,
                    "checksum": "b04fb4f4f3ab83b9d1b9f6f9eb236f1c04a9fd61bef7cee16b12df8ac911766a"
                }
            }
        },
        "post_processing_size": 30720045,
```

Good point. Should we anticipate already that we may add other fields in the future and change the code to support the addition of new fields without breaking backward compatibility in the future?

I added:
- post processing features (inside a PostProcessedInfo object)
- backward compatibility for dataset info
- post processing tests (as_dataset and download_and_prepare) for map (change features), select (change number of elements) and add_faiss_index (add indexes)
And I fixed a bug in `map` that I found thanks to the new tests

Now I just have to move `post_processing_resources_checksums` to  PostProcessedInfo as well and everything should be good :)
Edit: done