Title: Allow downloading/processing/caching only specific splits
Author: albertvillanova (MEMBER)
State: open
Created: 2021-04-22T17:51:44Z
Labels: enhancement

Allow downloading/processing/caching only specific splits without downloading/processing/caching the other splits.

This PR implements two steps to handle only specific splits:
- it allows processing/caching only specific splits into Arrow files
- for some simple cases, it allows downloading only specific splits (which is more intricate as it depends on the user-defined method `_split_generators`)

This PR makes several assumptions:
- `DownloadConfig` contains the configuration settings for downloading
- the parameter `split` passed to `load_dataset` is just a parameter for loading (from cache), not for downloading

Comments:
> If you pass a dictionary like this:
> 
> ```
> {"main_metadata": url_to_main_data,
> "secondary_metadata": url_to_sec_data,
> "train": url_train_data,
> "test": url_test_data}
> ```
> 
> then only the train or test keys will be kept, which I feel not intuitive.
> 
> For example if the users asks to load the "train" split, then the main and secondary metadata won't be downloaded.
> You can fix that by keeping all the keys except the splits to ignore

Hi @lhoestq, I have been thinking about this and I think it is worth that we discuss about it.

When I created this PR, my first idea was to create a "hack" inside the download manager that will be able to filter some split(s) without touching any dataset script. Of course, the download manager does not know about splits logic, and thus this trick would only work for some very specific datasets: only the ones containing that pass a dict to the download manager containing only the keys "train", "validation", "test" (or the one passed by the user for advanced users knowing they can do it), e.g. the `natural_questions` dataset (which was one of the targets).

The big inconvenient of this approach is that it is not applicable to many datasets (or worse, it should be constantly tweaked to cope with exceptional cases). One exceptional case is the one you pointed out. But I see others:
- the split keys can be different: train, test, dev, val, validation, eval,...
- in `hope_edi` dataset, the split keys are: TRAIN_DOWNLOAD_URL, VALIDATION_DOWNLOAD_URL
- in `few_rel` dataset, the split keys are: train_wiki, val_nyt, val_pubmed,..., pid2name
- in `curiosity_dialogs`, the split keys are: train, val, test, test_zero; this means that for every split we pass, we will also get test_zero
- in `deal_or_no_dialog`, each of the splits URL is passed separately to the download manager, so all splits would be always downloaded
- etc.

Then after discussing, another idea emerged: pass a `split` parameter to `_split_generators`, which know about the splits logic, so that it can handle which splits are passed to the download manager. This approach is more accurate and can be tweaked so that it works with all the datasets we want. The only inconvenient is that then for every target dataset, we must modify its corresponding `_split_generators` script method.

My point is that I don't think it is a good idea to implement both approaches. They could even interfere with each other! 

If you agree, I would implement ONLY the second one, which is simpler, more consistent and stable and will avoid future problems.

Hi @albertvillanova !
Yup I agree with you, implementing the 2nd approach seems to be the right solution