Title: Add checksums
Author: lhoestq (MEMBER)
State: closed
Created: 2020-04-29T13:37:29Z

### Checksums files

They are stored next to the dataset script in urls_checksums/checksums.txt.
They are used to check the integrity of the datasets downloaded files.
I kept the same format as tensorflow-datasets.
There is one checksums file for all configs.

### Load a dataset

When you do `load("squad")`, it will also download the checksums file and put it next to the script in nlp/datasets/hash/urls_checksums/checksums.txt.
It also verifies that the downloaded files checksums match the expected ones.

You can ignore checksum tests with `load("squad", ignore_checksums=True)` (under the hood it just adds `ignore_checksums=True` in the `DownloadConfig`)

### Test a dataset

There is a new command `nlp-cli test squad` that runs `download_and_prepare` to see if it runs ok, and that verifies that all the checksums match. Allowed arguments are `--name`, `--all_configs`, `--ignore_checksums` and `--register_checksums`.

### Register checksums

1. If the dataset has external dataset files

The command `nlp-cli test squad --register_checksums --all_configs` runs `download_and_prepare` on all configs to see if it runs ok, and it creates the checksums file.
You can also register one config at a time using `--name` instead ; the checksums file will be completed and not overwritten.

If the script is a local script, the checksum file is moved to urls_checksums/checksums.txt next to the local script, to enable the user to upload both the script and the checksums file afterwards with `nlp-cli upload squad`.

2. If the dataset files are all inside the directory of the dataset script

The user can directly do `nlp-cli upload squad --register_checksums`, as there is no need to download anything.
In this case however, all the dataset must be uploaded at once.

--

PS : it doesn't allow to register checksums for canonical datasets, the file has to be added manually on S3 for now (I guess ?)

Also I feel like we must be sure that this processes would not constrain too much any user from uploading its dataset.
Let me know what you think :)

Comments:
Looks good to me :-) 

Just would prefer to get rid of the `_DYNAMICALLY_IMPORTED_MODULE` attribute and replace it by a `get_imported_module()` function. Maybe there is something I'm not seeing here though - what do you think? 

> * I'm not sure I understand the general organization of checksums. I see we have a checksum folder with potentially several checksum files but I also see that checksum files can potentially contain several checksums. Could you explain a bit more how this is organized?

It should look like this:
squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

In checksums.txt, the format is one line per (url, size, checksum)

I don't have a strong opinion between `urls_checksums/checksums.txt` or directly `checksums.txt` (not inside the `urls_checksums` folder), let me know what you think.


> * Also regarding your comment on checksum files for "canonical" datasets. I understand we can just create these with `nlp-cli test` and then upload them manually to our S3, right?

Yes you're right

Update of the commands:

- nlp-cli test \<dataset\> : Run download_and_prepare and verify checksums
  * --name \<name\> : run only for the name
  * --all_configs : run for all configs
  * --save_checksums : instead of verifying checksums, compute and save them
  * --ignore_checksums : don't do checksums verification

- nlp-cli upload \<dataset_folder\> : Upload a dataset
  * --upload_checksums : compute and upload checksums for uploaded files

TODO:
- don't overwrite checksums files on S3, to let the user upload a dataset in several steps if needed

Question:
- One idea from @patrickvonplaten : shall we upload checksums everytime we upload files ? (and therefore remove the upload_checksums parameter)

Ok, ready to merge, then @lhoestq ?

Yep :)