Title: ADD S3 support for downloading and uploading processed datasets
Author: philschmid (CONTRIBUTOR)
State: closed
Created: 2021-01-12T07:17:34Z

# What does this PR do?

This PR adds the functionality to load and save `datasets` from and to s3. 
You can save `datasets` with either `Dataset.save_to_disk()` or `DatasetDict.save_to_disk`. 
You can load `datasets` with either `load_from_disk` or `Dataset.load_from_disk()`, `DatasetDict.load_from_disk()`. 

Loading `csv` or `json` datasets from s3 is not implemented. 

To save/load datasets to s3 you either need to provide an `aws_profile`, which is set up on your machine, per default it uses the `default` profile or you have to pass an `aws_access_key_id` and `aws_secret_access_key`. 

The implementation was done with the `fsspec` and `boto3`.


### Example `aws_profile` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
```

</details>


### Example `aws_access_key_id` and `aws_secret_access_key` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key", 
                            aws_secret_access_key="fake_secret_key"
                           )

load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk",
                            aws_access_key_id="fake_access_key", 
                            aws_secret_access_key="fake_secret_key"
                           )
```

</details>

If you want to load a dataset from a public s3 bucket you can pass `anon=True` 

### Example  `anon=True` :

<details>

```python
dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

load_from_disk("s3://moto-mock-s3-bucketdatasets/sdk",anon=True)
```

</details>

### Full Example

```python
import datasets

dataset = datasets.load_dataset("imdb")
print(f"DatasetDict contains {len(dataset)} datasets")
print(f"train Dataset has the size of: {len(dataset['train'])}")

dataset.save_to_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")

remote_dataset = datasets.load_from_disk("s3://moto-mock-s3-bucket/datasets/sdk", aws_profile="hf-sm")
print(f"DatasetDict contains {len(remote_dataset)} datasets")
print(f"train Dataset has the size of: {len(remote_dataset['train'])}")
```

Related to #878 


I would also adjust the documentation after the code would be reviewed, as long as I leave the PR in "draft" status. Something that we can consider is renaming the functions and changing the `_disk` maybe to `_filesystem`

Comments:
I created the documentation for `FileSystem Integration for cloud storage` with loading and saving datasets to/from a filesystem with an example of using `datasets.filesystem.S3Filesystem`. I added a note on the `Saving a processed dataset on disk and reload` saying that it is also possible to use other filesystems and cloud storages such as S3 with a link to the newly created documentation page from me. 
I Attach a screenshot of it here. 
![screencapture-localhost-5500-docs-build-html-filesystems-html-2021-01-19-17_16_10](https://user-images.githubusercontent.com/32632186/105062131-8d6a5c80-5a7a-11eb-90b0-f6128b758605.png)